Skip to content

node: keep trusted keys the control plane still lists (#916) - #917

Open
p1c2u wants to merge 2 commits into
google:mainfrom
whiteredhouse:fix-trusted-key-pruning
Open

p1c2u wants to merge 2 commits into
google:mainfrom
whiteredhouse:fix-trusted-key-pruning

Conversation

@p1c2u

@p1c2u p1c2u commented Nov 18, 2026 •

Copy link
Copy Markdown

A trusted key's age counted from when the node first received it, and a /keys sync kept that time. Pruning keeps only the newest key past the grace period, so a key the control plane still lists in its rotation grace period was dropped a day after the node first saw it, even when it signed the node's own identity. The next start then failed the role check ("biscuit: invalid signature") on a valid identity, and if the control plane could not be reached at that moment, both recoveries failed too.

  • mergeTrustedKeys: every key a /keys answer lists counts as vouched for at that sync, so pruning only drops keys the control plane stopped listing, or ones the node could not confirm for longer than the grace period.
  • pruneTrustedKeys: keeps every key with the latest time, since a /keys answer vouches for all its keys at once and its order says nothing about which is current, and never drops the key that signed the node's current identity.
  • NewAgentMeshNode: the configured key re-added next to persisted keys gets a fixed old time (the zero time is not stored and read back as now), so it cannot outrank the control plane's newer keys after a restart.
  • Start returns *StaleIdentityError when the stored identity fails the role check and neither recovery renews it. It unwraps to the role, refresh and recovery errors, so callers can tell a refusal (*RefreshError with its status) from a control plane that never answered.

Fixes: #916

A trusted key's age counted from when the node first received it, and a
/keys sync kept that time. Pruning keeps only the newest key past the
grace period, so a key the control plane still lists in its rotation grace
period was dropped a day after the node first saw it, even when it signed
the node's own identity. The next start then failed the role check
("biscuit: invalid signature") on a valid identity, and if the control
plane could not be reached at that moment, both recoveries failed too.

- mergeTrustedKeys: every key a /keys answer lists counts as vouched for
  at that sync, so pruning only drops keys the control plane stopped
  listing, or ones the node could not confirm for longer than the grace
  period.
- pruneTrustedKeys: keeps every key with the latest time, since a /keys
  answer vouches for all its keys at once and its order says nothing about
  which is current, and never drops the key that signed the node's current
  identity.
- NewAgentMeshNode: the configured key re-added next to persisted keys gets
  a fixed old time (the zero time is not stored and read back as now), so
  it cannot outrank the control plane's newer keys after a restart.
- Start returns *StaleIdentityError when the stored identity fails the role
  check and neither recovery renews it. It unwraps to the role, refresh and
  recovery errors, so callers can tell a refusal (*RefreshError with its
  status) from a control plane that never answered.
@p1c2u
p1c2u requested a review from aojea as a code owner November 18, 2026 20:02

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the trusted key management and pruning logic. It simplifies key merging by updating the received time for all active keys, introduces a structured StaleIdentityError for better error unwrapping, and ensures that the key signing the node's own identity is preserved during pruning. Additionally, it prevents configured fallback keys from outranking newer keys. The review feedback identifies style guide violations in the new tests where errors from ed25519.GenerateKey are ignored; these should be resolved by using the mustGenerateKey(t) helper.

Comment thread internal/node/node_test.go Resolved
Comment thread internal/node/node_test.go Resolved
Use mustGenerateKey instead of ignoring ed25519.GenerateKey's error, and
report a failed store close (errcheck).

@aojea aojea left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the report and the fix. The bug is real. On main, with the real functions: enrollment gives the node the current key, the first /keys sync adds the retiring key with a later time, and after 24 hours pruning keeps only the retiring key.

The fix for #916 is smaller than this PR. Please reduce the PR to that fix.

Keep

  1. The sync gives each key in the /keys answer the time of the sync.
  2. Pruning keeps all the keys that have the latest time, not only the first one.

Change

  1. Remove mergeTrustedKeys. It does not merge now, and existing has no effect. Put the loop in syncTrustedKeys.
  2. Add a log line for each key that pruning drops, so that the node log shows the cause the next time.
  3. Correct the text that still has the old meaning ("first learned"):
    • receive_time of TrustedSigningKey in api/agentmesh.proto. Then run make proto, and npm ci in sdk/js followed by ./hack/gen-sdk-proto.sh.
    • The --key-grace-period row in site/content/docs/reference/agentmesh-node.md.
  4. Tests: one subtest for the sequence of the issue (sync, then prune) and one row in TestPruneTrustedKeys.

Remove from this PR

  1. The keep parameter and signedOwnIdentity.
  2. neverVouched.
  3. StaleIdentityError.

The inline comments give the reason and the code for each item. With these changes applied on main, go test -race ./internal/node/ ./sdk/meshnet/ and make lint pass, and the two new tests fail without the fix.

Proposed text for item 5:

  // When the control plane last vouched for the key: the last /keys answer
  // that listed it, or the enrollment or rotation event that announced it. A
  // member that prunes drops a key a grace period after this, unless no key
  // has a later time; unset means unknown and is read as now.
  google.protobuf.Timestamp receive_time = 2;
| `--key-grace-period` | `24h` | How long the node keeps a control plane key after the control plane last confirmed it in a `/keys` answer or a rotation event. The keys of the latest confirmation always stay. A key that `/keys` no longer lists is removed at the next sync. |

Comment on lines 88 to 99
// mergeTrustedKeys replaces the stored trust set with the authoritative set
// from /keys, preserving ReceivedAt for keys already known so local grace
// pruning keeps working across syncs.
// from /keys. The control plane lists every key that is still valid, its
// current key and any in their rotation grace period, so each listed key's
// ReceivedAt becomes now: local pruning then ages a key from the last time
// the control plane vouched for it, and never drops one it still lists.
func mergeTrustedKeys(existing []TrustedKey, fetched []ed25519.PublicKey, now time.Time) []TrustedKey {
merged := make([]TrustedKey, 0, len(fetched))
for _, key := range fetched {
tk := TrustedKey{Key: key, ReceivedAt: now}
for _, old := range existing {
if bytes.Equal(old.Key, key) {
tk.ReceivedAt = old.ReceivedAt
break
}
}
merged = append(merged, tk)
merged = append(merged, TrustedKey{Key: key, ReceivedAt: now})
}
return merged
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

existing has no effect now, and the function does not merge. Please delete this function and TestMergeTrustedKeys, and give the keys their time in syncTrustedKeys:

	// The answer lists every key the control plane still accepts, so it
	// vouches for each of them now: pruning ages a key from the last answer
	// that listed it, not from the first time the node saw it.
	now := time.Now()
	synced := make([]TrustedKey, 0, len(keys))
	for _, key := range keys {
		synced = append(synced, TrustedKey{Key: key, ReceivedAt: now})
	}

syncTrustedKeys still reads existing, to verify the answer.

Comment thread internal/node/node.go
Comment on lines +135 to +159
// StaleIdentityError is Start's error when the stored identity fails the
// role check against every trusted key and neither recovery renewed it. It
// unwraps to the role check's error and to both recovery errors, so a caller
// can tell a control plane that refused the identity (a *RefreshError with
// its status) from one that never answered.
type StaleIdentityError struct {
Role string
RoleErr error
RefreshErr error
RecoveryErr error
}

func (e *StaleIdentityError) Error() string {
return fmt.Sprintf("loaded identity fails role requirement %q (autonomous refresh: %v; refresh-token recovery: %v): %v", e.Role, e.RefreshErr, e.RecoveryErr, e.RoleErr)
}

func (e *StaleIdentityError) Unwrap() []error {
var errs []error
for _, err := range []error{e.RoleErr, e.RefreshErr, e.RecoveryErr} {
if err != nil {
errs = append(errs, err)
}
}
return errs
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#916 does not need this type, and nothing in this PR reads it. Please remove it here, together with its test, and send it in a separate PR with the code that uses it.

Comment thread internal/node/node.go
Comment on lines +521 to +529
// The configured key is only as fresh as the enrollment or /info
// answer that stored it. Next to persisted keys it must not count as
// just vouched for, or it would outrank the control plane's newer
// keys when pruning picks the newest; alone, it is all there is.
vouched := neverVouched
if len(trustedKeys) == 0 {
vouched = time.Now()
}
trustedKeys = append(trustedKeys, TrustedKey{Key: cfg.ControlPlanePubKey, ReceivedAt: vouched})

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a different problem from #916. LoadMeshConfig returns the first stored key, so the node comes here only when --control-plane-public-key gives a key that is not in the stored set. The change also writes the epoch as a special value into the persisted MemberCredential, which the SDKs load too.

Please remove it from this PR, with neverVouched and its test. If you can reproduce the problem, open an issue with the steps.

Comment thread internal/node/node.go
Comment on lines +2025 to +2038
// pruneTrustedKeys drops keys the control plane has not vouched for within
// gracePeriod, as when it stopped listing them or the node could not reach
// it, but always keeps the most recently vouched-for ones: the current
// signing key has no successor until a rotation event arrives, so ageing it
// out would empty the trust set and leave the node unable to verify any peer
// or router. A /keys answer vouches for all its keys at once and says
// nothing about which is current, so every key with the latest time is kept,
// never one of them by its place in the list. It also keeps every key keep reports, which is the key that
// signed the node's own identity: the control plane lists all its valid keys
// alike, so the newest vouched-for key need not be the one a refresh signed
// with, and dropping that one after a long time offline leaves a valid
// identity that fails the check at the next start, when only the control
// plane can renew it.
func pruneTrustedKeys(keys []TrustedKey, now time.Time, gracePeriod time.Duration, keep func(ed25519.PublicKey) bool) []TrustedKey {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove the keep parameter and signedOwnIdentity:

The loop below stays as it is, without keep. The comment becomes shorter, for example:

// pruneTrustedKeys drops the keys the control plane last vouched for more
// than gracePeriod ago, but always keeps the ones with the latest time: a
// node that cannot reach the control plane must not age out its whole trust
// set and lose the means to verify any peer or router. A /keys answer vouches
// for all its keys at once and its order does not say which is current, so
// the latest time keeps all of them, not the first one in the list.
func pruneTrustedKeys(keys []TrustedKey, now time.Time, gracePeriod time.Duration) []TrustedKey {

In startKeyPruning, log each key that pruning drops:

				n.keysMu.Lock()
				before := n.trustedKeys
				pruned := pruneTrustedKeys(before, time.Now(), gracePeriod)
				changed := len(pruned) != len(before)
				n.trustedKeys = pruned
				snapshot := append([]TrustedKey(nil), pruned...)
				n.keysMu.Unlock()
				for _, tk := range before {
					if !containsTrustedKey(pruned, tk.Key) {
						logger.Infow("[KeyPruning] dropped control plane key", "key", fmt.Sprintf("%x", tk.Key), "last_vouched", tk.ReceivedAt.UTC().Format(time.RFC3339), "grace_period", gracePeriod.String())
					}
				}

Comment on lines +369 to +374
// TestKeyPruningKeepsTheIdentitySigner covers a node whose identity was
// signed by a key it learned before the newest one it holds, as when the
// control plane's newest key reached it earlier or its stored key was added
// again at a restart: pruning must not drop the signer, or the next start
// finds a valid identity that no trusted key verifies.
func TestKeyPruningKeepsTheIdentitySigner(t *testing.T) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the reduced change, two tests are sufficient. Both fail on main.

  1. A subtest in TestSyncTrustedKeys for the sequence of the issue:
	// The node got the current key at enrollment and the retiring key from a
	// later sync. While /keys lists both, pruning must keep both, whatever
	// the time since the node first saw each (#916). A key that /keys no
	// longer lists is dropped at the sync.
	t.Run("a key /keys still lists outlives the grace period", func(t *testing.T) {
		expiredPub, _ := mustGenerateKey(t)
		srv := keysServer(t, []ed25519.PublicKey{retiredPub, currentPub}, []ed25519.PrivateKey{retiredPriv, currentPriv})
		defer srv.Close()
		n := newNode(t)
		enrolled := time.Now().Add(-25 * time.Hour)
		n.trustedKeys = []TrustedKey{{Key: currentPub, ReceivedAt: enrolled}, {Key: expiredPub, ReceivedAt: enrolled}}
		if err := n.syncTrustedKeys(context.Background(), srv.URL); err != nil {
			t.Fatalf("syncTrustedKeys: %v", err)
		}
		if containsTrustedKey(n.trustedKeys, expiredPub) {
			t.Error("a key /keys does not list stayed in the trust set")
		}
		got := pruneTrustedKeys(n.trustedKeys, time.Now(), 24*time.Hour)
		if len(got) != 2 || !containsTrustedKey(got, retiredPub) || !containsTrustedKey(got, currentPub) {
			t.Errorf("pruning kept %d keys, want the retired and the current key that /keys lists", len(got))
		}
	})
  1. A row in TestPruneTrustedKeys. Add it at the end, because the table sets tests[3].want by index:
	sameSync := k(48 * time.Hour)
		{"every key of the latest sync survives", []TrustedKey{stale, staler, sameSync}, []TrustedKey{stale, sameSync}},

Please remove TestKeyPruningKeepsTheIdentitySigner, TestKeyPruningKeepsEveryLatestKey, TestFallbackKeyStaysOldThroughTheStore and TestStaleIdentityErrorUnwraps together with the code that they test.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Node prunes its identity’s signing key while /keys still advertises it

2 participants